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Abstract. Motivated by the enumeration of a class of plane partitions studied by Proctor 
and by considerations about symmetry classes of plane partitions, we consider the problem of 
enumerating lozenge tilings of a hexagon with "maximal staircases" removed from some of its 
vertices. The case of one vertex corresponds to Proctor's problem. For two vertices there are 
several cases to consider, and most of them lead to nice enumeration formulas. For three or 
more vertices there do not seem to exist nice product formulas in general, but in one special 
situation a lot of factorization occurs, and we pose the problem of finding a formula for the 
number of tilings in this case. 



1. Introduction and statement of results 

The study of lozenge tilings is warranted by the many useful insights they bring in the 
subject of plane partitions. Some important instances of these are presented in [18] and [6]. 
In this paper we present some more such connections. 

A plane partition is a rectangular array of nonnegative integers with the property that all 
rows and columns are weakly decreasing. A plane partition contained in an a x b rectangle 
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and with entries at most c can be identified with its three dimensional diagram — a stack 
of unit cubes contained in an a x b x c box — , which in turn can be regarded as a lozenge 
tiling of a hexagon H(a, b, c) with side lengths a, b, c, a, b, c (in cyclic order) and angles 
of 120° (see Figure 1.1 and [8] or [23]; a lozenge tiling of a region on the triangular lattice 
is a tiling by unit rhombi with angles of 60° and 120°, referred to as lozenges). This 
simple bijection is the crucial link between the theory of lozenge tilings and that of plane 
partitions. For example, the number of tilings of H(a, b, c) follows to be equal to the number 
of plane partitions that fit in an a x b x c box, which is, by a result due to MacMahon [20], 
UtiIlUUUi(i + j + k- l)/(i + j + *-2). 

As a variation of this, Proctor [22] considered the problem of enumerating those plane 
partitions ir contained in an a x b x c box for which the projection of tt on one of the coordinate 
planes, say on Oxy, fits in the "maximal staircase" A = (b, b — 1, . . . , b — a + 1) (when A is 
viewed as the corresponding Ferrers diagram) contained in the ax b basis rectangle (we are 
assuming here, without loss of generality, that a < b; see Figure 1.1 for an example of such 
a plane partition with a = 5, b = 8, c = 3). Proctor [22] found that this number is given by 
the simple product 



n 



n ! ' +1 c + i + j - 1 b yj l 2c + i + j - 1 

3=1 J j=b-a+2 J 



(1.1) 



By the above bijection, it is easily seen that Proctor's problem is equivalent to counting the 
lozenge tilings of H(a, b, c) with a maximal "staircase of lozenges" removed from a corner 
at which edges of lengths a and b meet (simply view Figure 1.1 as being two dimensional; 
there, a maximal staircase of lozenges was removed from the southeastern corner). 

What if we require that the projection of n on two of the coordinate planes be contained 
in the corresponding staircases? The above bijection shows that the question is equivalent 
to counting the number of tilings of H(a, b, c) with two maximal staircases removed, from 
vertices that are non-adjacent and non-opposite (this is illustrated in Figure 1.2; there, 
maximal staircases were removed from the southeastern and the western corner; the plane 
partition is the same as in Figure 1.1). There are six cases to consider, corresponding to the 
six relative orderings of a, b and c. These are shown in Figures 1.3(a)-(f). (At this point, 
the special marks in form of ellipses should be ignored.) Mirror reflection pairs up these six 
cases in three pairs — the rows of Figure 1.3. 

We draw all the hexagons H(a,b, c) and the regions obtained from them by cutting corners 
so that the horizontal edges have length 6, and the other two pairs of parallel edges, as we 
move counterclockwise, have lengths a and c. Let H<{(a,b,c) be the region obtained from 
H(a,b,c) by removing maximal staircases from the northwestern and eastern corners (the 
subscript stands for the "diagonal" position of the cut off corners). 

For a region R on the triangular lattice, denote by L(R) the number of its lozenge tilings. 
In the special case when R is obtained from a hexagon H(a, b, c) by removing staircases 
from two of its corners, we define three more weighted tiling enumerators for R as follows. 
Consider the tile positions that fit in the indentations of the zig-zag cut that removed a 
staircase of lozenges (the possible such positions are marked by ellipses in Figure 1.3(c)). 
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Figure 1.2. 




Figure 1.3(e). ff d (3,5,8). 



Figure 1.3(f). tf d (3,8,5). 



By weighting these tile positions by 1/2, one creates a new, weighted count of the tilings 
of R: each tiling T gets weight l/2 fc , where k is the number of lozenges of T occupying 
positions weighted by 1/2, and the sum of weights of all tilings T of R gives the weighted 
tiling enumeration 1 . Clearly, one can choose to weight by 1/2 only the tile positions along 
the cut that removed the northwestern corner of H(a,b,c), or, furthermore, to weight by 
1/2 only the tile positions along the cut that removed the eastern corner. These three 
possibilities define our three weighted enumerators. We denote them by L*, L*, and L„, 
where a superscript (respectively, subscript) indicates weighting along the cut from the 
northwestern (respectively, eastern) corner. 

If b < c < a (see Figure 1.3(a)) — or, by mirror reflection, c < b < a (sec Figure 1.3(b)), 
— neither L(H^(a, 6, c)), nor the weighted enumerators L*(H^[a, b, c)), L*(_ffd(a 7 b, c)), and 
Lt(Hd(a,b,c)) seem to be given by simple product formulas. The other two cases lead to 
the following results. 

Theorem 1.1. Ifb<a<c (see Figure 1.3(c) for an example), we have 

L(H d (a, b, c)) = (-l) b ^' 2 P b (a - 2b - 1, b + c + 1), (1.2) 
where P n (x,y) denotes the product on the right hand side of (1.8). 

Note. All the factors in Pb(a — 2b — 1, b + c + 1) arc positive except for the factors in the 
shifted factorial (a — 36 — c + 2j — which are all negative since for the largest factor in 
this product we have 

a - 3b - c + 3j - 2 < a - c - 2 < -2, 

as a < c in the case addressed by Theorem 1.1. Therefore, for a, b and c as in Theorem 1.1, 
the sign of P b (a - 2b - 1, b + c + 1) is (-l) fc ( b+1 )/ 2 , which checks that the right hand side of 
(1.2) is non-negative. 

lr The motivation to consider such weightings comes from the fact that weightings of that kind arise 
whenever the Factorization Theorem from [2] is applied to a (symmetric) region on the triangular lattice. 
See [2, Sec. 6] [3] [4] [5] [6] [9] [11] and the proof of Theorem 1.4 in Section 2 for examples. 
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Still keeping the relative order b < a < c of the parameters, the weighted enumerators 
L*(£/d(a, b, c)) and L*(Hd(a, b, c)) do not seem to be given by simple product formulas. But 
there is one for £*(i/d(«, b, c)). 

Theorem 1.2. Ifb < a< c (see Figure 1.3(c) for an example), we have 
L:(H d (a,b,c)) 

= 2 _„_6 TT (j - 1)! (a + g - b + 2j - 1)! (c - a + 3j - ljfe-j (a + 2c + 3j - l) fc _ J+1 
1 = 1 (6 + c + j - 1)! (a - & + 2j - 1)! ^ 

w/iere i/ie shifted factorial (a)k is defined by (a)k := a(a + 1) • • • (a + k — 1), fc> 1, and 
(a) := 1. 

For a < 6 < c, plain enumeration of the tilings of Hd(a, c) does not seem to be given 
by a simple product formula. There is, however, a simple formula for L*(H<i(a,b,c)). As 
we are going to show in Section 2, the following result follows easily from a determinant 
evaluation of the second author [16, (5.3)]. 

Proposition 1.3. If a <b < c (see Figure 1.3(e) for an example), we have 

T*(H(nhA\ 1 TT H - 1)! (fc + 2c - 3a + 3j) (b + c - 2a + j)\ 
L ( Hd (a,b,c)) = ¥ [[ ( c _ a + 2 ,-l)! 

(b + 2c - 3a + 2j + l)j_i (26 + c - 3a + 2j + l)j_i 
(6 -a + 2. 7 -l)! • 

Note. The special cases when c = a — 1 or when & = a — 1 form the subject of [6, Proposi- 
tion 2.2]. 

The weighted enumerator L* clearly makes sense also for the region H\(a, b, c) obtained 
from H(a, b, c) by cutting off just the northwestern corner. We have the following counter- 
part of Proctor's formula (1.1). 



Theorem 1.4. For a < b, we have 



L"(H 1 {a,b,c)) = ^f[ 



n'"' a c + i + j-1 -pi- 2c + i + j - 1 
i + j — 1 A A i + j'-l 



(1.4) 



What if we require that the projection of it on all three coordinate planes be contained in 
the corresponding staircases? An example illustrating this, using the same plane partition 
as in Figures 1.1 and 1.2, is shown in Figure 1.4. Clearly, by the bijection between plane 
partitions and lozenge tilings, plane partitions satisfying these conditions are identified with 
tilings of the region H^(a, 6, c) obtained from H(a, b, c) by removing maximal staircases from 
three alternating vertices. No matter what the relative ordering of the side lengths a, b and 
c is, there are always two staircases that "interfere" — i.e., there is no portion of an edge 
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Figure 1.4. 



Figure 1.5. T 5 . 



of the hexagon H(a, b, c) separating them. Data suggests that there are no simple product 
formulas in general. (Note that, among the cases with two removed staircases, the relative 
orders of a, b, c not covered by Theorems 1.1, 1.2 and Proposition 1.3 are precisely those in 
which the staircases interfere.) 

However, the special case a = b = c presents significant factorization. Indeed, letting 
T a stand for the triangular region H^{a,a,a) (the case a = 5 is shown in Figure 1.5), the 
number of tilings of T a factors as follows for a < 7: 



The amount of factorization is remarkable (also for larger a; we have computed and factored 
L(T a ) up to a = 30) and comparable, say, to that of the numbers enumerating domino tilings 
of squares (see [15]). Based on this, we pose the following problem. 

Problem 1.5. Find a formula for the number of lozenge tilings of T a , that explains the 
large amount of prime factorization of these numbers. 

Let us now go back to the case of plane partitions with the property that their projection 
on two coordinate planes is contained in the corresponding staircases. As we saw, in terms 
of tilings this amounts to counting the tilings of a hexagon with maximal staircases removed 
from two non-adjacent and non-opposite vertices. What if we remove them from adjacent 
vertices? This leads to the region H a (a,b,c) (illustrated in Figure 1.6; evidently, the sub- 
script stands for "adjacent" ), obtained from H(a, b, c) by removing maximal staircases from 



L(T 2 ) 
L(T 3 ) 
L(T 4 ) 
L(T 5 ) 
L(T 6 ) 
L(T 7 ) 



2 



3 2 

2 3 • 13 
2 2 • 5 2 • 31 
2 • 3 2 • 19 2 • 37 
2 • 7 3 • 13 • 43 • 127 
2 7 • 3 5 • 5 3 • 7 • 13 • 73. 
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Figure 1.6. £T a (3,8,5). 



the top two corners (we assume that these two staircases do not interfere — otherwise the 
leftover region has no lozenge tilings; non-interference amounts to b > a + c — 1). 

The H a (a,b,c)'s form a family of regions that resemble the case b < a < c of the 
H<i(a,b,cys (compare Figures 1.6 and 1.3(c)). Even though they are different, it turns 
out that the enumeration of tilings of both families of regions reduces to the evaluation of 
the same determinant, the one in Theorem 1.10. We have the following result. 

Theorem 1.6. For b > a + c — 1, we have 



L(H a (a, b, c)) = P a (b - a, c), 

where P n (x,y) is the product on the right hand side of (1.8). 

Also here, let us consider weighted enumerators. As before, let us denote by £*, L*, and 
£» the weighted tiling enumerators where a superscript (respectively, subscript) indicates 
that the tiles along the northwestern zig-zag line (respectively, northeastern) are weighted 
by 1/2. While L*(H a (a, b, c)) and L*(H a (a, b, c)) do not seem to be given by simple product 
formulas, this is the case for L*(_ff a (a, b, c)). 

Theorem 1.7. For b > a + c - 1, we have 



x 



n 



(a + b-c+2) a 

(j - 1)! (b + c - a + 2j - 1)! (b - a - c + 2j + 2), (b + 2c-a + 3j + 2) a _ 



(b + 2j-l)\(c + a-j)\ 



In view of the previously made observation that in the case that b < a + c—1 the maximal 
staircases interfere and their removal results in a region which is not tilablc, it may seem 
absurd to insist on having "analogues" of Theorems 1. 6 an 1.7 for b < a + c — 1. But why, 
instead of removing maximal staircases, not remove partial staircases? To be precise, if 
a + b + c = mod 2, then let us remove the partial staircase (a — 1, a — 2, . . . , (a — b + c)/2) 
from the top-left vertex of the hexagon, and the partial staircase (c— 1, c— 2, . . . , (c— b+a)/2) 
from the top-right vertex. (See Figure 1.7(a), in which the removed staircases are indicated 
by the white regions. The shades should be ignored at this point.) We obtain a region 
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(a + 6-c)/2 




(c + 6-a)/2 



& 



Figure 1.7(a). 



Figure 1.7(b). ff„(6,5,7) 



that looks like a pentagon with an "artificial" peak glued on top. Any lozenge tiling of this 
region is uniquely determined in the rhombus that is composed out of the triangular peak 
and its upside-down mirror image. (In Figure 1.7(a) this rhombus is shaded, and the unique 
way to tile this rhombus is shown.) Hence, we may equally well remove this rhombus. The 
leftover region now has the form of a pentagon with a notch (see Figure 1.7(b); at this point 
the ellipses are without relevance). Let us denote this region by H n (a,b,c). Remarkably, 
extensive computer calculations suggest that the number of lozenge tilings of H n (a, b, c) is 
given by a "simple" product formula. We state it as Conjecture A.l in the Appendix. The 
fact that the result, even though given in terms of a completely explicit product, is unusually 
complex 2 may indicate that proving the conjecture may be a formidable task. 

Moreover, it seems that the region H n (a, b, c) also allows a weighted enumeration which 
is given by a simple product formula. Let us, as before, denote by L*, L* , and L» the 
weighted tiling enumerators where a superscript (respectively, subscript) indicates that the 
tiles along the northwestern zig-zag line (respectively, northeastern) are weighted by 1/2. (In 
Figure 1.7(b) these tiles are marked by ellipses.) While L*(H n (a, b, c)) and L*(H n (a, b, c)) do 
not seem to be given by simple product formulas, this seems to be the case for L%(H n (a, b,c)). 
We state it as Conjecture A. 2 in the Appendix. Again, the result is unusually complex, which 
may indicate that a proof may be a considerable undertaking. 

There is one more possibility for choosing two corners of the hexagon from which to 
remove maximal staircases — two opposite corners. Data suggests that in general this does 
not lead to simple product formulas. There is one exception, when the sides supporting the 
removed staircases are equal (see Figure 1.8 for an example), but this is a "semi-frozen" 

2 No nontrivial simplifications seem to be possible. We are not aware of any other "nice" result which is 
similarly involved. 
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Figure 1.8. # (4,6,4). 




Figure 1.9. Figure 1.10. 



situation — each tiling decomposes in tilings of parallel strips of width two (see the proof 
of Proposition 1.8 in Section 2). 

Let H {a,b,c) be the region obtained from H(a,b,c) by removing maximal staircases 
from the western and eastern corners. (Not unexpectedly, here, the subscript stands for 
"opposite.") 

Proposition 1.8. L(H (a, b, a)) = (b + l) a . 

A different viewpoint that can naturally lead one to consider the regions introduced above 
(hexagons with corners cut off) is based on symmetry classes of plane partitions. (It was in 
fact this viewpoint that provided the original motivation to study these regions.) Consider 
the regular hexagon H 2n '■= H(2n,2n,2n). The ten symmetry classes of plane partitions 
contained in a cube of side 2n (see [24] for their definition) are identified with the ten 
symmetry classes of tilings of H 2 n (see [18]). Define a ray of tiles to be a sequence of n tiles 
extending from the center of ti 2 n to the nearest point of one of its edges. The six rays of tiles 
of Hm are shown, for n = 3, in Figure 1.9. It is easy to see that the tilings of Hi n that have 
CSTC symmetry (i.e., cyclically symmetric, transposed complementary tilings) contain the 
tiles of all six rays, and the restriction of such a tiling to one of the six congruent regions 
left by removing the rays determines the whole tiling uniquely (see Figure 1.9). The regions 
H ( i(a, b,c), b < a < c, form a two parameter generalization of this (compare Figures 1.9 and 
1.3(c)). 

Similarly, TC (i.e., transposed complementary) symmetry forces inclusion of two opposite 
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Figure 1.11. 



Figure 1.12. 



rays in the tiling, and reduces to enumerating tilings of one of the two pieces left over after 
removing the two opposite rays of tiles (see Figure 1.10). The regions corresponding to 
the plane partitions considered by Proctor (hexagons with one corner cut off) form a one 
parameter generalization of this (compare Figures 1.10 and 1.1). 

We are thus naturally led to consider the regions generated by removing three alternating 
rays, as shown in Figure 1.11. This region does not correspond to a symmetry class of plane 
partitions, but it is nevertheless quite compelling to consider in this context. The regions 
H a (a,b,c) form a two parameter generalization of it (compare Figures 1.11 and 1.6). The 
two parameters were essential in conjecturing a formula for the number of tilings of these 
regions, based on data: polynomials fully factored into linear factors contain much more 
information than just integers factored into small primes. 

To finish this analysis, we mention that removing one ray (see Figure 1.12) leads to a 
region whose number of tilings has a simple product expression, as it easily follows from the 
Factorization Theorem of [2] and the results of [4] . 

To rephrase the above statements, if one regards the regions formed by removing rays as 
being built up of 60° sectors, the tilings of the regions consisting of 1, 2, 3 or 6 sectors are 
enumerated by simple product formulas. Data suggests that this is not the case for 4 or 5 
sectors. 

We prove our results in Section 2 by employing a standard bijection between lozenge 
tilings and non-intersecting lattice paths, thus, due to the Lindstrom-Gessel-Viennot the- 
orem [19] [12], obtaining a determinant for the number (respectively weighted count) of 
tilings that we are interested in, and, finally, by evaluating the resulting determinants. In 
most of the cases, we obtain special cases of the two determinant evaluations that we state 
in Theorems 1.9 and 1.10 below. Theorem 1.9 is due to the second author [16, (5.3)] (see 
[1] for a simple proof, which is reproduced in [6]). The determinant evaluation in Theo- 
rem 1.10 does not seem to have appeared previously in the literature. The paper [7] contains 
our original proof, which is rather involved, but has its own appeal as it contains a non- 
automatic (!) application of Gosper's algorithm [13] (see also [14, Sec. 5.7] [21, Sec. II. 5]). 
Later we discovered that, in fact, there is a combinatorial argument which transforms the 
determinant in Theorem 1.10 into an instance of the determinant in Theorem 1.9, so that 
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these two determinant evaluations are actually equivalent. It is this argument that we give 
in Section 2. 



Theorem 1.9. Let x, y and n be nonnegative integers with x + y > 0. Let K n (x,y) be the 
matrix 



K n (x,y) := 



(x + y + i+j- 1)! 
( x + 2i-j)\(y + 2j-iy.J 1 < ij < n 

x + y + i + j - 1 N 



x + 2i-j\ y + 2j-i ))x<ij< n 



(1.5) 



Then 



detfK (x v)) 17 {] 1)1 (X + V + J)l {2X + V + 2J + (X + 2V + 2J + fl 6) 

det^^y))- jyL (x + 2. ? -l)!( ? y + 2. ? -l)! ' U ' bj 

where, as before, the shifted factorial (a)k is defined by (a)k ■= a(a + 1) • • • (a + k — 1), 
k > 1, and (a)o := 1. 

Theorem 1.10. Let n be a positive integer, and let x and y be nonnegative integers. Let 
A n (x,y) be the matrix 

Then 

j-Ji {x + n + 2j)\(y + n- j)\ 

Note. Theorems 1.9 and 1.10 are only formulated for nonnegative integral x and y. But in 
fact, with a generalized definition of factorials and binomials (cf. [14, §5.5, (5.96), (5.100)], 
both theorems would also make sense and be true for complex x and y. 

2. Proofs of the results 

As already mentioned at the end of the Introduction, we employ in our proofs a standard 
bijection that maps each lozenge tiling T of a region R on the triangular lattice to a family 
of non-intersecting lattice paths taking steps east or north on the grid lattice Z 2 . This 
bijection works as follows. Choose a lattice line direction d — without loss of generality, 
the horizontal direction — , and mark with a dot the unit segments parallel to d on the 
boundary of R (see Figure 2.1). Each marked segment is either on top or at the bottom of 
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(a - 2j+ 2, c+j-l) 




Figure 2.3(a). Figure 2.3(b). 



the region R. Label from right to left the ones at the bottom of R by u\, 112, ■ . ■ , u m and 
the ones on top of R by v\ , Vi , . . . , v n . 

Consider now the tile t\ of T resting on u i} for some fixed 1 < i < m (sec Figure 2.2 
for an example of a tiling T of the region from Figure 2.1). Let ti be the other tile of T 
containing the side of t\ opposite Mj. Continue the sequence of selected tiles by choosing 
ts to be the tile of T sharing with £ 2 the side of t 2 opposite the one common to t 2 and t\ . 
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This procedure leads to a path of rhombic tiles growing upward, which clearly must end 
on one of the v/s. This path of rhombi can be identified with a (linear) path that starts 
at the midpoint of Uj and ends at the midpoint of Vj, see Figure 2.2. (There, the resulting 
linear paths are indicated by dotted segments.) After normalizing the oblique coordinate 
system and rotating it in standard position, we obtain a lattice path on Z 2 that starts at 
the midpoint of Ui (actually, its image after these normalizations), ends at the midpoint of 
Vj (again, actually its image after these normalizations) and takes unit steps east or north. 
(See Figure 2.3(a) for the resulting lattice paths in our example.) One obtains this way a 
family V of m lattice paths, one for each 1 < i < m, and they cannot touch each other since 
the corresponding paths of tiles are disjoint. We obtain in particular that m < n, and hence 
by symmetry m = n. It is easy to see that the correspondence T i— » V is a bijection between 
the set of tilings T of R and the families V of non-intersecting lattice paths starting at the 
midpoints of u\, U2, ■ ■ ■ , u n , ending at the midpoints of v\, V2, ■ ■ ■ , v n and contained within 
R. 

Lemma 2.1. Forb< a < c, we have L(H d (a,b,c)) = (-l)&(*>+i)/2 dct A b (a-2b- 1, b+c+1). 

Proof. Label the horizontal unit segments on the boundary of H d (a,b,c) as described 
above. Choose an oblique coordinate system with the origin at the midpoint of u\ and coordi- 
nate axes parallel to the non-horizontal lattice lines of the triangular lattice (see Figure 2.1). 
Applying the procedure described above to H d (a,b,c) one obtains that L(H d (a,b,c)) is 
equal to the number of families of non-intersecting lattice paths with starting points Ui — 
(—i + 1, i — 1), i = 1,2, ... ,b, ending points Vj = (a — 2j + 2, c + j — 1), j = 1, 2, . . . , b, and 
with the additional requirement that they do not touch the line y = x — 2. (This require- 
ment ensures that the corresponding paths of rhombi stay within our region H d (a, b, c); see 
Figure 2.3(a). Note that by abuse of notation we denote the midpoints of the Ui's and Vj's 
by the same symbols we use for the segments.) 

By the Lindstrom-Gessel-Viennot theorem [19, Lemma 1] [12] [25, Theorem 1.2], the num- 
ber of such families of non-intersecting lattice paths is given by the determinant of the 6x6 
matrix whose (i, j)-entry is the number of lattice paths from Ui to Vj that are strictly above 
the line y = x — 2. By Andre's Reflection Principle [10], this number is 

a + c - j + 1\ _ / a + c- j + 1 
v a - 2j + i + l) \a - 2j - i + 1 

Therefore, the Lindstrom-Gessel-Viennot theorem implies that 

L(H d (a,b,c)) = d C t(( a + '- J + \)-( a+ r J + 1 )) ■ (2-1) 

Reversing columns (i.e., replacing j by 6 + 1 — j) in (2.1) the right hand side becomes 
(-l^-D/Wf a + c + J- b )_( a + c + j-b \\ 



a + 2j + i - 26 - V \a + 2j - i - 26 - IJ J 1 < tJ < b 

The entries of the matrix in (2.2) are readily seen to be precisely the negatives of the entries 
of Ai,(a — 26 — 1, 6 + c + 1). This implies the statement of the Lemma. □ 
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Proof of Theorem 1.10. The preceding proof of Lemma 2.1 shows (by renumbering the 
lattice paths from left to right) that (— l) n ^ n+1 ^ 2 A n {x, y) counts the number of families V 
of n non-intersecting lattice paths, with starting points (— n + i,n — i), i = 1,2, ... ,n, and 
end points (x + 2j + 1, y — j — 1), j = 1, 2, . . . , n, with the additional requirement that they 
do not touch the line y = x — 2 (see Figure 2.3(a) for an example). Now, for each such 
family, we prepend (2n — 2i + 1) vertical steps to the i-th path. Thus we obtain families V' 
of n non-intersecting lattice paths, with starting points (— n + i, —n + i — 1), i = 1,2, ... ,n, 
and end points (x + 2j + 1, y — j — 1), j = 1,2,. . . , n, with the additional requirement that 
they do not touch the line y = x — 2. (See Figure 2.3(b) for the resulting path family in our 
example.) In fact, this is a bijection between the former and latter path families, because 
the prepended path portions arc in fact forced by the boundary y — x — 2. Therefore 
(—l) n ( n+1 )/ 2 A n {x, y) is also equal to the number of the latter path families. 

Again we may apply the Lindstrom-Gcsscl-Vicnnot theorem. Since the Reflection Prin- 
ciple yields that the number of paths from (— n + i, —n + i — 1) to (x + 2j + 1, y — j — 1) 
which do not touch the line y = x — 2 is given by 

x + y + j +2n-2i + l\ _ fx + y + j + 2n - 2i + 1 
x + 2 j + n — i + 1 J \ x + 2j + n — i 

{y-x- 3j) (x + y + j + 2n-2i + l)\ 
~ (x + 2j + n - i + 1)! (y + n - j - i + 1)! ' 

we infer that (-l)™ (n+1)/2 A n (x, y) is equal to 

det f (y-x-3j){x + y + j + 2n-2i + l)\ 



( x + 2j + n-i + iy.(y + n-j-i+ 1)\J 



J]_(y -x-3j)- det 



1 fx + y + j + 2n-2i+l 



}J(y - x - 3.]) ■ det 



y + n - j - i + 1 V x + 2j + n - i + 1 / / ^ .< n 
(_l)*+2j+n-i+i f-y-n + j + i- 1 N 



J=1 x y + n-j-i + l\x + 2j + n-i + l // 

n 

U((-ir +n - j (y-x-3j)) 

3 = 1 

1 /—y — n + j + i — 2 



x det 



-y - x - 2n + 2i - j - 2 \x + 2j + n - i + 1 



l<i,j<n 



(At the second to last equality we used that (^) = (— l) fe ( _ " H fe fe_1 )-) This latter determinant 
is the determinant det K n (—y — x — 2n — 2, x + n + 1), with K n {x, y) defined in (1.5). In 
view of Theorem 1.9, this proves (1.8), after some manipulation. □ 

Proof of Theorem 1.1. This follows directly from Lemma 2.1 and Theorem 1.10. □ 

Proof of Theorem 1.2. As in the proof of Lemma 2.1, we map the tilings to families V of 
non-intersecting lattice paths, with starting points = (— i + l,i — 1), i = 1, 2, . . . , b, ending 
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points vj = (a— 2j+2, c+j—1), j = 1, 2, . . . , b, and with the additional requirement that they 
do not touch the line y = x—2 (see again Figure 2.3(a)). Following the proof of Theorem 1.10, 
we prepend 2i — 1 vertical steps to the i-th path (because here we kept the numbering of the 
paths from right to left), so that we obtain families V' of non-intersecting lattice paths, with 
starting points Ui = (—i + 1, —i), i = 1, 2, . . . , b, ending points Vj = (a — 2j + 2, c + j — 1), 
j = 1, 2, . . . , b, and with the additional requirement that they do not touch the line y = x — 2 
(see again Figure 2.3(b)). However, unlike in the proofs of Lemma 2.1 and Theorem 1.10, 
here each family has a certain weight, given by the £-th power of 1/2, where £ is the number 
of lozenges that are weighted by 1/2 in the corresponding tiling (compare Figures 1.3(c), 
2.1 and 2.3(b)). 

To realize this weight, we give each horizontal step from (a — 2k + 1, c + k — 1) to 
(a — 2k + 2, c + k — 1), for some k, the weight 1/2. This takes care of the fact that the 
lozenges along the northwestern zig-zag line are weighted by 1/2. To also take into account 
that the lozenges along the eastern zig-zag line are weighted by 1/2, we assign a weight of 
2 (sic!) to each horizontal step from (fc, k) to (k + 1, k), for some k (i.e., to each horizontal 
step which terminates directly at the line y = x — 1 the paths are not allowed to cross). 
This generates our weight, up to a multiplicative constant of 2 a . Indeed, for each marked 
lozenge position along the eastern boundary, a tiling T has a lozenge in that position if and 
only if the corresponding unit segment weighted by 2 is not a step on some lattice path of 
the family corresponding to T, thereby giving rise to a missing weight of 1/2 in comparison 
to path families where some path does contain that step. To give an explicit example, the 
tiling in Figure 2.2 contains two tiles weighted by 1/2 along the eastern boundary, the third 
and the fifth from the bottom. These are the ones which are completely white, as there is 
no lattice path running through them. Hence, in the corresponding path family shown in 
Figure 2.3(b), the step from (2, 2) to (3, 2) and the step from (4, 4) to (5, 4) (both weighted 
by 2) are not contained by any of the paths. 

Now we want to write down the Lindstrom-Gessel-Viennot determinant for our weighted 
count (as defined just above). In order to do so, we need the weighted count of paths from 
(— i + 1, —i) to (a — 2j + 2, c + j — 1) which do not touch y = x — 2. We claim that the 
weighted count of these latter paths is the same as the weighted count of all paths from 
(— i + 1, —i) to (a — 2j + 2, c + j — 1), in which the last step of the path has weight 1/2 if it 
is a horizontal step. (It should be noted that in this weighted count there are no steps with 
weight 2 anymore.) This is seen as follows: suppose we are given a path from (— i + 1, — i) 
to (a — 2j + 2, c + j — 1) which does not touch the line y = x — 2, and which has exactly 
I touching points on the line y = x — 1. These £ touching points on y = x — 1 must be 
reached by horizontal steps from (k, k) to (k + 1, k), each of which contribute a weight of 
2. Thus, in total, this gives contribution of 2 l to the weight. Now we map such a path to 
2 l paths from (— i + 1, —i) to (a — 2j + 2, c + j — 1) (without restriction), by focussing on 
the path portions between two consecutive touching points, including the portion between 
(— i + 1, — i) and the first touching point, and keeping any of them either fixed or reflecting 
it in the line y — x — 1. This proves the assertion. 

By distinguishing between the cases where the last step of a path is vertical respectively 
horizontal, the new weighted count of the paths from (—i + 1, —i) to (a — 2j + 2, c + j — 1) 
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Figure 2.4. 



then is seen to be 

fa + c + 2i - j - 1\ 1 /o + c+ 2i - j - 1\ _ 1 (a + 2c + 3i - 1) (a + c + 2i - j - 1)! 
\ a-2j + j + l /2V a-2j + i )~2 (a - 2j + i + 1)! (c + i + j - 1)! 

Therefore the Lindstrom-Gessel-Viennot determinant is seen, by manipulations similar 
to those in the proof of Theorem 1.10, to be 

/ 1 (a + 2c + 3t - 1) (a + c + 2z - j - 1)! \ 
6 V2 (a-2j + i + l)!(c + i+j-l)! A<„-< fc 

- n (m +2c+3t-i» det („ ,) ) isusi . 

After reversing the order of rows and columns (i.e., after replacing z by 6+1 — -i and j 
by b + 1 — j), it is seen that this determinant is the determinant detKf,(— a — c — b — 
1, a — b). Application of Theorem 1.9, division of the resulting expression by 2 a , and some 
rearrangement finish the proof of the theorem. □ 

Proof of Proposition 1.3. Consider the region H<±(a, b, c) and choose the direction d in the 
bijection between tilings and lattice paths to be the direction of the lattice lines going from 
southwest to northeast (see Figure 2.4). Consider the unit segments parallel to d on the 
boundary, and label the midpoints of those on the eastern boundary, from top to bottom, by 
ui, i*2, . . . , u a , and the midpoints of those on the northwestern boundary by v\, V2, ■ ■ ■ , v a - 
Choose an oblique coordinate system centered V3 units above u\ (see Figure 2.4) and with 
axes along the northwestern and western lattice line directions. 

By the bijection between tilings and lattice paths, each tiling T of H t \(a 1 b, c) is identified 
with a family V of non-intersecting lattice paths with starting points (— 2i, i), i = 1, 2, . . . , a, 
and ending points (c — a — j,b— a + 2j), j = 1,2,..., a. (Unlike in the proof of Lemma 2.1, 
no additional requirements on the paths are needed). 
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Figure 2.5. R(5, 10, (7,6,4,2,-1)). 

The weighted enumerator L* assumes that the northwestcrnmost a tile positions are 
weighted by 1/2 (these positions are marked in Figure 2.4). Correspondingly, the paths 
of V whose last step is a vertical step have weight 1/2 (as before, the weight of a lattice 
path is the product of the weights of its steps). The weight of the family V is the product 
of the weights of its members, and by construction it matches the weight of the tiling T. 
By the Lindstrom-Gessel-Viennot theorem, the total weight of those families V that are 
non-intersecting — and hence L*(-ffd(a, 6, c)) — is given by the determinant of the a x a 
matrix whose (i,j)-entry equals the total weight of the paths from Ui to vj. Splitting the 
latter family in two according to the type of the last step, one obtains that its total weight 
is 

(b + c - 2a + i + j - l\ 1 (b + c - 2a + i + j - l\ 
V c - a + 2i-j-\ / 2 \ c-a + 2i- j ) 

_b + 2c-3a + 3i (b + c - 2a + i + j - 1) ! 

~ 2 (c - a + 2i - j) ! (b - a + 2j -i)V 

When computing the determinant of the above matrix, the j-frec factors can be factored 
out along rows, and the leftover matrix is precisely the one in (1.5) with n = a, x = c — a, 
and y — b — a. Using (1.6) and substituting the values of x and y one obtains the formula 
in the statement of Proposition 1.3. □ 

Now we turn our attention to the proof of Theorem 1.4. We deduce (1.4) from a well- 
known determinant evaluation (sec the proof of Lemma 2.2) and Proctor's formula (1.1), 
using the Factorization Theorem for perfect matchings of [2] . 

Let m and n be nonnegative integers, and let 1 = (h,fo, ■ ■ ■ , In), < h < m, h > ■ ■ ■ > l n 
be a list of integers (some of which may be negative). We define the regions R(n, m, 1) 
as follows. Consider an oblique coordinate system on the triangular lattice, centered at 
the midpoint of a lattice segment facing northeast and having the x-axis horizontal and 
the y-axis parallel to the lattice lines going from southwest to northeast (see Figure 2.5). 
Consider, on the one hand, the points (— i + 1, i — 1), i = 1, 2, . . . , n, and on the other hand 
the points (lj,m — lj), j = 1,2, ... ,n. We construct R(n, m, 1) so that its tilings are in 
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bijection with the families of non-intersecting lattice paths with these starting and ending 
points. It is easy to see that this determines R(n, m, 1) to be the hexagon with side lengths 
n, l\, m — li, l\ — l n + 1, l n + n — 1, m — l n — n + 1 (in anticlockwise order, starting with the 
southwestern side), and having triangular dents along the northeastern side at the lattice 
segments with midpoints not among (lj, m — lj), j = 1, 2, . . . , n. (R(5, 10, (7, 6, 4, 2,-1)) is 
shown in Figure 2.5.) 



Lemma 2.2. 



L(R(n,m,\)) 



III 



<i<j< 



n (U-lj)U:=i(m + i-l)\ 



Ilti(h + n-iy.utM-hy. ■ 

Proof. By construction and by the Lindstrom-Gessel-Viennot theorem, we have that 



L(R(n, m, 1)) = det 



m 



(2.3) 



l<ij<n 



This determinant can be evaluated, e.g., by means of [17, (3.12)] with A = m — 1 and 
Li = li — 1, i = 1, 2, . . . , n, and one obtains the formula in the statement of the Lemma. □ 

Proof of Theorem 1.4- Consider the region H\{a,b,c) and weight by 1/2 the a tile 
positions required to be weighted so by L* (these are marked in Figure 2.6). Draw a 
line / through the centers of the marked lozenges, and reflect H\(a,b,c) across I. The 
union U of H\(a,b,c) with its mirror image is precisely the region R(2c, a, 1), where 1 = 
(b, b — 1, . . . , b — c + 1, a — c, a — c — 1, . . . , a — 2c + 1). Applying Lemma 2.2 one obtains, 
after some manipulations, that 



b — a 

n 



c + i + j - 1 



n 

j=b-a+l 



2c + i + j - 1 



(2.4) 



The region U is symmetric about I, so we can apply to it the Factorization Theorem of [2] 
(see [5] for a phrasing of it in terms of lozenge tilings). Following the prescriptions in the 
statement of this theorem, cut U along the zig-zag line following lattice segments just above 
I (this is shown as a thick line in Figure 2.6), and denote the pieces above and below the 
cut by U + and U~, respectively. In U~ , weight the tile positions just below the cut by 1/2. 
Since I cuts through 2a unit triangles, the Factorization Theorem yields 



L(U) = 2 a L(U+)L* (£/"). (2.5) 

However, U~ is by construction just H± (a, 6, c). Moreover, in U + there is a row of forced tiles 
(shaded in Figure 2.6), and the region left upon their removal is congruent to Hi(a, b— 1, c). 
Solving for L*(U~) in (2.5) and using formulas (2.4) and (1.1), one obtains for L*(H\{a, b, c)) 
the product expression (1.4). □ 

Even though the region H a (a, b, c) looks different from the case b < a < c of H~d(a, b, c), 
it turns out that their tiling enumerations amount to evaluating the same determinant. 
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Figure 2.6. 



Figure 2.7. 



Lemma 2.3. For b> a + c — 1 we have L(H a (a, b, c)) = det A a (b ~ a, c). 

Proof. Rotate the region H a (a,b,c) clockwise by 60°, so that it is positioned as in Fig- 
ure 2.7. By the bijection between tilings and lattice paths, each tiling is identified with a 
family of non-intersecting lattice paths with starting points (— i + l,i — 1), ending points 
(c — j + 1, b — a + 2j — 1), i, j = 1, 2, . . . , a, and such that all lattice paths stay strictly 
above the line y = x — 2. Just as in the proof of Theorem 1.1, the Lindstrom-Gessel-Viennot 
theorem implies that 

K V " \\b-a + 2j-ij \b-a + 2j + i)) 1 < ij < a 

The above determinant is readily recognized as A a (b — a, c). □ 

Proof of Theorem 1.6. This follows directly from Lemma 2.3 and Theorem 1.10. □ 

Proof of Theorem 1.7. We use the bijection between tilings and families of non- intersect- 
ing lattice paths from the proof of Lemma 2.3. In addition, we prepend (2i — 1) vertical steps 
to the i-th path. Thus we obtain families V' of non-intersecting lattice paths, with starting 
points (— i + 1, — i), i = 1, 2, . . . , a, ending points (c — j + 1, b — a + 2j — 1), j = 1, 2, . . . , a, 
and such that all lattice paths stay strictly above the line y = x — 2. For the rest of the 
proof one follows the arguments in the proof of Theorem 1.2, which have to be adjusted 
only insignificantly. □ 

Proof of Proposition 1.8. Let T be a tiling of H Q (a, b, a). Consider the b tiles containing 
the lattice segments on the bottom part of its boundary. Because of forcing, there is precisely 
one dent in the upper boundary of the union of these tiles (see Figure 2.8). This dent has 
to be covered by some other tile t (shaded dark in Figure 2.8), which in turn forces b more 
tiles in place. Thus, a subregion congruent to H(l,b,l) at the bottom of H Q (a,b,a) ends 
up being tiled by the restriction of the tiling T. Since there are b + 1 tilings of H(l, b, 1) 
(corresponding to the b + 1 possible positions of t) , this implies 



L(H a (a, b, a)) = (b + 1) L(H Q (a - 1, 6, a - 1)). 
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Figure 2.8. 

Repeated application of this gives the statement of the Proposition. □ 

Appendix 

Conjecture A. 1 . The number of lozenge tilings of the region H n (x, m + y,x + m — y) (see 
Figure 1.7(b) for an example) is equal to 

-pr (x + i)\ yt" (x + 2m-i + l)\ 



*-_L(x-i + m + y + l)\ (2i - 1)! i _^ 1 fl (2m + 2y - 2i + 1)! (to + x - y + i - 1)! 

m-l 3/— 1 

x 2(?)+oo jj ^!^ ^! ^( :E+^+ i)™- 2^ -l^( a; "2 /+ i +3^ )L*-f j-2 

i=l j=i i>o j>0 

x 11^ + 3 ~f v + \ 3 4 1 + iHfHt 1-2 11^ + ¥ - w + LIJ + 2 % Htj 

i>0 j>0 



x HV* 1 L2J 1 " 1 ^)1\\\- m -2i 11V- L2J 1 " 1 ">m-2\ \ \-2i-2 

y mi" 1 

jQ(x - y + 3« + l) m +2y-4i ] [ (x + 771 - y + i + l) 3 y —m—4i 
i=0 i=0 



i>0 



y 

T[(x + i + 2) 2m -2i-l 



X 7 7 x • (A.1) 

(x + y + 2) m _j / _i (to + x - y + l) m+y 

Here, shifted factorials occur with positive as well as with negative indices. The convention 
with respect to which these have to be interpreted is 

a(a + 1) • • • (a + k - 1) if k > 0, 
(n),, J 1 ifk = 0, 

l/(a- l)(a-2)---(a + fc) i/fc<0. 
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All products Yli> (f(i)) g (i) in (A.l) have to interpreted as the products over all i > for 
which g(i) > 0. 

For a proof one could try to proceed as follows: first we introduce nonintersecting lattice 
paths, with starting points along the bottom, and end points along the northeastern and 
northwestern zig-zag lines. On introducing a suitable coordinate system, the starting points 
can be represented as Ai = i = 1,2,.. . ,m + y, and the end points as Ei = (x — i, 2i), 

i = 1,2, ... ,m, Ei = (m + y — 2i + 1, to + x — y + i), i = m + 1, m + 2, . . . , m + y. The 
corresponding Lindstrom-Gessel-Viennot determinant is 

■<«<•»+» V \ (JSZiSl,) « = '»+! m + v\) 

The task is to evaluate this determinant. In principle, after having taken suitable factors out 
of the determinant (so that the new determinant is a polynomial in x), the "identification 
of factors" method, as described in Section 2.4 of [17], should be capable of accomplishing 
the determinant evaluation. 

Conjecture A. 2. The weighted count of lozenge tilings of the region H n (x, m+y, x+m—y), 
where the lozenges along the two zig-zag lines are weighted by 1/2 (see Figure 1.7(b) for an 
example; the lozenges that are weighted by 1/2 are marked by ellipses), is equal to 

m t i ■ 1M m+y , ... 

(x + ^ - 1)! -pj (x + 2m-i)\ 

(„ _ A I ™ I ,. I 1 \| (OA _ i \\ 11 



L (x- i + m + w + l)!(2i-l)! 11 (2m + 2y - 2i + 1)! (m + x - y + i - 1)! 

i— m+1 

m-1 y— 1 

x 2(™)+(S) JJ i! JJ t! JJ(x + * + §) m - 2i _i U(a?-y + 3i + |)psj t _^-|_ 4 

i=l j=l i>0 j>0 

+ ¥ - y + Lf J + Dsm-Lf j-i no* + ¥ - » + rn + ^j-m 

i>0 i>0 

x JJ(x + m- LfJ+i + 1) 2L4J _ ro _ 2 , JJ^ + Lf J + * + 2 L-2Lfj -2,-2 
(a; - y + \)\j^^ +2y {x + m- y) y+1 JJ(x + i + l) 2m -2i 



i=0 



v ni- 1 

JJ(x - ?/ + 3i + 1) Q (x + m - y + i + 1) 3 j, 

— m—4i 



ry/21-1 

(m + x - y) m+y+1 (x + y + [f])^j_ y+1 JJ (x-y + l + 3i) 

i=0 

1 

" ILo^ + f - I + * + + f - ! + * + |)y-2i-i ^ 
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(A.3) 



with the same conventions as in the previous conjecture. 

For a proof, one could again introduce nonintersecting lattice paths, with starting points 
and end points as before. The corresponding Lindstrom-Gessel-Viennot determinant is 

det \\ {x-i+jy.{2i-j)\ i-L,...,m i 

l<i,j<m+y \ ^ 7 ; „. i - i -i 'it/ — F — i - a, ' i = m + 1, . . . ,m+ V I 

The remarks after (A. 2) apply also here. 
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